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Abstract 

For the last Three years the Military Training Centers OUC) 
PLATO (support group li^ss been experimenting with the development of a 
number of fortaallzed techniquee^ for assisting autUors in writing instruc- 
tionally effective courseware. These methods*, collectively referred to 
as ^^lesson reviewing'*, vary widely depending on individual authors' 
needs. Basically, however, a lesson review is a set of comments about the 
lesson ranging from alternative instructional strategy suggestions to 
grammatical corrections, from content accuracy to punctuation (see 
les?;on Review , MTC report /'3, 1976, tarry Francis, Merle Goldstein, and 
Eileen Call-Himwlch for examples and complete descriptions of review 
techniques) . 

This report describes the evolution of lha lesson review process 

and discusses review effectiveness in tenns of resulting lesson revision 

and author training. The report also details ^ number of recommendations 

fox improving review effectiveness in the future. The following areas: 

— reviewer characteristics 

— reviewer/author relationship 

— review content/format 

are emphasised. Some specific recommendations are that the reviewer 

should be on-jsite, that the reviewer's actual or perceived authority 

should be equal to the author* s, and that the reviewer and author should 

reach either a contractual or lesson formalized agreement about the purpose 

of the review and about what specific feedback each expects. 
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Preface 

Due to the personal iiaLui/e of critiquing lessons, .data £or tht* 
loUowing report is largely ^necdotal. Some information hAS .a^o btvn 
gathered through the questionnaire in Appendix A or i-n face to face or 
phone interviews.* During the cou oi our "reviewing" experlnenti^ and 
evaluation work, we have observed a marked inconsistency between what 
authors say 'and what they do* Some authors replied with nothing but 
praise both to review^ themselves and to the questionnaire, but had 
rarely responded to anything more than the most rudimentary recommendations 
in reviews* This dichotomy proved to be our nemesis for qufte a time 
before we finally began hearing the ''raetatalk" underneath the words — 
rfhat was really being communicatpd rather than the words that were beinft 
said* 
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latroductioa 

For tiie laJ^L three years the MilUary Training Centers (rTTC) 
support group has been ^sxperimenting with the development of a number of 
formalized techniques for assisting authors in writing course^^are that is 
more instruct tonally effective. These methods, collectively referred to 
as "lesson reviewing", vary widely depending on individual authors' 
needs- Basically^ however, a lesson review is a set of comments about the 
les$on ranging from alternative instructional strategy suggestions to 
graniTTiatical corrections, from content accuracy to punctuation- 

Lessoa reviewing was originally conceived for the dual purpose of: 
1. aiding authors in lesson revision 
2+ training authors in instructional "design. 

The purpose of the following report is to both describe the evolution 
of the lesson review process and discuss review effectiveness in terms of 
lesson revision and author training. 
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Lesson Reviews 

t 

Ratio nale " 

Whon the >n'C group fin^t begau reviewing lessonSj the rationale 
behind the need for rtiviews was grounded in a threefold problem. First j 
while a n^^mber of ARPA authors had teaching experience^ few were familiar 
with either computer-based education or instructional design* Many 
authDr^ fouad th^ transition from classroom to computer both perplexir^g 
and frustrating. l-Jhat once could have been taught in a lecture format 
now required a more innovative approach. Instructors who could once 
comfortably rely on standardized course objectives and lists were suddenly 
wrestling with the subtleties of student behavioral objectives and credible 
criterion testing. In addition, since the >rrc group offered a two to 
three week course In both the TUTOR language and the rudiments of 
instructional strategy, many author5J felt somewhat harried over having 
to learn TUTOR, instructional design, and the mechanics of the PLATO 
keyset and editor all at one tiine- Part of that frustration was the 
sometimes stated* more often implied conviction that instrucrional 
design seminars "interrupted" the ''more important'' task of learning 
TUTOR. Having trained a number of diverse author groups, we've found 
tlie premise that instructional strategy Is^basically intuitive aad that 
authors need little if any assistance In lesson desfgn to be ^ particu- 
larly persistent, widely held attitude. This is especially true of former 
instructors w*io sometimes feel that knowledge of one medium implies 
knowledge of another medium^ i.e. success in a lecture situation implies 
''knowing how to teach'' using an approach even as dissimilar and indivi- 
dualized as CBE. 
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objt-cLivc ippr.iL^ai ot LtK-ir work, p.ir t icul ar ly in rh^^^t.- artais n.^t 
roadily analyztiHK^ i rem stud^^iit ^lertoriuaiKi.* tiatcj , .moviiiK <it iEUi."r" 

at:t ioii, orv;-ini :;'it ion, n Iternati appro^aches^ vL.stial prt^^ientat ion^ <*Lc . 

Lesson reviewing, th^ut^h by no mean^; i^ieal* s^eemed a |)^>Lent La 1 ly 
effective w^iv ot handiirtg all three situations- 

Evoltrtion ut" Lhe Keviewin^ PreL<^et>s 

^^^51|!^^ Ct'A ^vI?.5^ reviews . Although ^^iC did a fe^^." earlv revievjs tiL 

t^iaiiuLe* the fir.^L formaiizad reviews wvre written tor the Aberdeen 

- * 

"^ivUMiinisc Cotjrse Project at Aberdeer. Proving; (;rounds. By the ent} ^ i the 

Abtrdeen project, approximately fifteen TTC-reviews liad been written. 

Tho^e ^arly review^;, later referred to as *\nd-of- Lesson** reviews, woro 

often a ^ -nbination ot codii^>i and ias^t rucL ioiia i design suggestions with 

the nain emphasis on the latter. Wliile the tone ana slant of e<ich review 

vrertainiy varie<* i th individual rt^viewrs, end^of "lesson reviews nW 

had a number of ^.t^mnon characteristics* 

1. They were alwav^^ written after the le^son had been (^onpleted. 

i. Tiiey wt>re njlways rt^ndored in writt<'n iorEr, Irt^quentJy 

Int^ludin^i an annotated printont oi th^^ lesst*n sMj*(:t'St inii 
^:odin>^ i^hange*** t ext ual rev i s ion=> K-it^ . 

3* Ihey gen^Lrally r^^'^uired Iromono to two munths^ r<. he writt^'U 
and del ivered to the .luthur. 
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liu* ^tibstiince ot end ot lesson reviews centered on ^iny or all of 
:our l**vfls ut" lo^son dt velonment ; 

— dti^i ^^tn 

— Implenentat icii 

Each level in turn spanned a variety of jjossible problem are.*;* and 
reeonm^ndat ions or ^.ij^j;estions :or alternatives* The foJlawing exjmples 
detail many ot the .sorts of is^sije:; that were dealt with. 
Level (>iie — Planning 

— intontioas, go;ili>, objectivt^s of the lesion 
""<is<;umptiuns aboijt students' enterin>; ability and knowlod^e 

of the subject (,e.g. terminology) 
— the relationship between the iriivldual lesson and 
the overfill curriculum 

— the relevance of Individual Lopirs ro the lesson as a whole 

Le ve I 1 wo — Ihrsi ^ n 

— rhoii^e of teaching stratej^ien^; 

— seley^t ion/ appropri a tellers of media 

— individualized rout inj^ 
— ori;a(iizat ion of i:oateni . 

LeVc-l Three — Imp lementnt ion/ Oevel opment 

— eff**ctive utilisation of teaching »traLe>^ies and nedium 
— cprtectlve feedback and remediation 
— appropriateney<> of tone and style 

reliabi 1 ity of the criterion test (if present) 

— transition from fram^* to fratfie, Lopir to topic 
— quality and quantity of stJidcnt Inter... tlon 
--r lar i ty of text 

— appropriatenetis of reading level, i 1 1 astrat ions* i-tc. 

Ia-vuI Kour — Polish 

--iirammar, spelling^ typographical errors, etr. 

— consiiJtency of terminology, instructions, keys, etc* 

— visual presentation (including textual layout) 

Reviews were generally formated to include the following elements; 
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1- ^ covt^r lettL'T onuTneritinji the rji}ov points hrojictit ujt 
in z)w body ot the review 

;?L-n\>rjl ^..*'^JTlont :^ and Mi/>;t'sti^J L*ian^ie>' 

l iowciurtii oT hoth t?ic pri SJrTiit U-^*^on ^strarturo jiid -^^ti;^twted - 
orj^ani^iW: tonal revisions 

4. an annotated printout of the program lt«;eif vlth I lnL*^hy-i ine 
textual or coding suggestions- 

For 3 complete description of Aberdeen lo^^iicn diiveiopti:><,*nt » revit^u^; and 

courseware see Summary and Analysis of Aberdoei^ CfiE Project (cal i-lUmwit:h , 

1977; HimwicK, 1977) 

While Aberdeen ^lutViors usually claimed that the t:oinments and j^uj-^f^est ions 
the/ received reviews were "useful*' and "helpful"* they also seemed 
inclin^id to simply file reviews away and forget about them. MTC 
r^viewrs conceded that Xhey generally had scant success with getting 
authors to make anything other than superficial lessou revisions (i»t7. 
correcting misspellings* etc.). In addition it was "ten difficult to 
get authors to comment about the reviews th<*msclves with anything tnort,* than 
a perfunctory "thank you". flTC reviewers felt that definite changes in 
reviewing needed to be made. The establishment of the Sheppa*'d project 
presented an excellent opportunity to experiment with alternative i»?view 
methods. 

Reviewing pitfallii . HanperL^d by ^ lack .>f author tL-edback, MTC 
reviewr^i weru forced to draw ihtMr own ( oiu hision*^' ^hout the new 
form or direction reviews should Lake. nu>y began by art^^ming that the 
quality of the reviews wat; not in <|uetition* but rather th<iL other morL' 
subjective or ambiguous elements wcr*^ Involved. The fact that authors 
were so reluctant to ta^k about revtewti seemed to support the as«;umption 



I 



7 



LhaL fot-*lln^s racher than is*5ues v.'ere at stake* Thus, HTC turned its 
attention from rovi^ing review comments to revamping review techniques , 
and iror^ the inipact rcvitsvs had oa lessons; to the effect they had on 
authors* 111 e!?phasi*,ing the subjective elements of revLewSj reviewers 
sec about trying; to pinpoint the possible technical or psychological 
pitfalls in effect at the time a lesson v'as revieu'ed * 

Psycti-olo^ical Elements * f'Vom>rrC's perspectives three psychological 
factors played heavily in the acceotance or rejection of a lesion review. 
Fifiit ^^nd most obvious was the problem of the author's Sjubjective involve- 
ment in the leeson. As one (male) author effusedj "Writing lessons is a 
very creative, difficult thing* It's like giving birth/* Thus, just as 
artists <iomt_*t imey find It difficult to separate themselves from their work, 
author*; otten find it difficult to separate themselves from their lessons. 
To^cT:Utc42'e the lessjon Mas to criticize the author. This was especially 
true if th^ autlior And reviewer had never met* In this instance the 
r*^viewer was often regarded more ^is an adversary than an allyj the review 
itself more as d threat than a tool* 

Apart from the question of subjectivity, there was the problem of 
time* A thorough end-of-le&son review typically required a number of 
wt-^ek^; to prep ire. During that time the author had usually begun work on a 
new lesson- Since enthusiasm for the old iesson was usually replaced by 
the preoccupation and momentum of work on the new le£;son4 revision was 
often Relegated to <^ linbo statu.^ to be Carried out "as soon aa thi.^ new 
lesson in finished". As the old lee son 'got "colder" j revision seetned 
less and less important until it was easiest to regard a lesson as 
''finished" simply because coding was complete* 
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ho^x^vi-^t waiJ tile probU^m of p:irL ver-u-- tV*o vJiole. An\* t<isV. nc^re 

1 

hurdeni-^om** viewed as a Vv-hoK* fcitber tiiaii j^^srts. Sitvly^ tM^^h 
oriLicIsm or riu^;^e^ti(>n a reviowt^r m;icJe i <i irave seetned rt^asonaMo. 
CoUeciively, however* tht* suggested ch.iiMt.^s ma have )>oon oTrerwiielninj^- 
After recc^ivinfi a printout annotated In red ink* one author ruefully 
' ubiierved, "It looks like it bl^d to death.'* He also chose to ij^nore it. 

Technical itlement The two major technical pro'blens we perceived 
centered around one outstanding dilemma**-time* Ac the ARPA sites we 
workeii with, authors were under considerable pressure Co meet semester 
datea^ project deadlines, etc* Thus, they sometimes felt an understand- 
.^i^ble reluctance to *'wastt^ time" revising le<*sons which were es;;entJ.Uly 
considered f i nished . 

In addition, reviewers sometiiaes foand thonsejves suggesting rhanftus 
which'would have required a Mjor overhaul* The n^eed for such substantial 
r^viiiion might well have been averted had the Author and reviewer .bron 

s 

able to consult in the planning stajjes, usini^ the review as a propn^al 
rather ti^an a post ntorteB. 

The problem of author/reviewer consultation was also a stumbliiig 
block. Almost by definition* an end-of-lc*if;on review wa.*; a lengthy* 
one-'sided excursion through the lusson'p strengths and shortcomings. Tills 
nonologue quality* coupled ^-itli XM.^ oCtier inliibitivt^ uleru^nts* prob^M ' di^i 
li^ttle to enhance any real e xchan j^e of . idt^is between author and rt-'. .L-v-.vr* 

In-pro^ress re^newing * With the onset of the ?>hepp^3rd Al'Si paripodiral 
project, >frC felt that the time w^is right tor exper iment in>5 with different 
review techniques* The main problem was to develop a flcxi'ble pVtr^^iijs tha^ 
could: 



11 



X. reduce author Jefensiveness by examining the lesson be fore 
the author's subjunctive Involvement became too strong 

2. capitalii^e on lesson momentum by giving the author immediate 
teedbdck while the lesson was stilJ "current" 

3. optimize author ti^mti by enabling the author to make any 
necessary modifications or corrections before tuey became - 
habitual. 

The most reasonable first step seemed to tie to h^gin reviewing lessons 
as they were being written^ in bits and pieces find at v£>,rious stages of 
develt>pment . To maintain continuity and an overfill perspective^ in-progress 
reviewing was intended as a cumulative (rather than fragmentary) approach 
in wlUch the reviewer would reexamine "old" sections of a lessc^n in 
addition to each successive "n^" Action. Since the reviewer had only to 
deal with part of a lesson at a time^ review comments could be written much 
more quickly. In order to provide immediate feedback we also established 
an on-line "review" file in v^iich reviewers wrote general comments, and 
autiier^ were encouraged to make replies or rebuttals, ask questions, etc. 
Additional more *;pecific conimentsi were then sent somewhat later (usually 
within the week^ in a hardt^opy form along with a lesion printout mainly 
noting suggest ions> for textual changes, alternative wordings, etc. The only 
problem was that sfnce the author w.is continually revising and expanding the 
lesson the ha^'dcopy comments were sometimes outdated by the time they 
reached him (her). 

The development, of PLATO inter-terininal commmiicat ion and monitoring 
capabilities' nlso added another dimens^ion to reviewing.' it was now 



This PLATO communications ^feature enables two people at two different 
terminals to go through a lesson "together". One person enters a lesson, 
presses certain keys* and the lesson appears simultaneously on the second 
terminal. Any responses entered into the first terminal also cause the 
d/isplay on the second terminal to react identically. Authors are also nble' 
to "talk" to each other on-line while in this monotoring frameworks 
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p. .wt>K^ ior t;k nu :^>r anJ the rc'viewer to b^ at dilU'ient ,^jres and yet 

vith i?acJi oLlU'j , Sitit'c Liu^ roviowi T u.^^i-iilv iiaJ >^pocin<: co'-rL-iiLs in nin<l 
dbuut a number ot ditiortiiit sections ot ihe lesson, ihti .luthor wjiu^illv 
monitoroj l!u Ti viewer k^j that iho reviewer could direct the i r prpr.r^^ss 
tlirou^h the lesson. This method not only made, it e^i,^ier for t-Mch to 
explain various <^otntnents or thouglits about tlu- lesson, but also enriched 
the review as inore au i-xciiange of ideas :and less a set. of di ret t Ives given 
over to the author, l^\il(i triciny authors favored thi^ method, it was ext^r^^tnely 
timt! consuming, r<irn;inii from l-^ to nearlv 3 hours in len>;th. and rocju'rin^ 
a good deal of nentai stcininci on the part of both the" aut!ior and t!ie 
rev iewer - 

f 

In-progress review,s covered essentially the same areas and levels as 
e[id-of-Jesson reviews, Wliyrea^ end-^f-lesson reviews tended to Ije more 
lesson oriented, reviewers tried to make in-progress reviews tnoi^e ju^^aor 
oriented, VJritten comment,^ were always prefaced with a recitation oi what 
tho reviewer had especially liked about the les.son, i,tr j»r<,piuci* 
iL -iiiic X are especially effective", or "Vou generally provide helpful 
tcedback for incorrect aniiwers", Revieu\irs also tried to intersperse 
ren:atning conrneOsLs with specific examples not only of what could be 
clianged but what was good. Wherever possible the reviewer tried to 
ri'ft.*rence one part of a losson and Ripply it as a solution to another part* 
e*:^. "This ?^eotion might be more easily iiluBLrated by tisir-j* ^tiit; same sort 
ol ^raph you used in st^cLion 

By th^' end ot the Sheppard paramedical i>roject, MTC rt-viewrs \\n<\ 
pertormed over 100 lesson review.s for Slieppard authors, For a thorough 
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description of^MTC review techniques with sample reviews see Lesson Review 
(Francis, (Joldstein, £i Call-Hlwich, 1975). 
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Conclusion 

Having expe rituentec! vith a number of reviewing approaches, we find 
oui'selVes ^ome sei\ses boggled by even more hypotheses than when we 
began^ As" is so often the case, the more one learns, the more there is 
to be learned. Our increasing understanding of the jsubtleties of working 
f^imultaneously oa an intellectual and emotional level has left us with 
iiKire questions than answers* Therefore, the following section enumerates 
not only what we know but what we don*t know* ■ 

The conclusions fall^ into two categories — what was accomplished and 
what might be accomplished* Topics in the first category include assesis- 
ments of review effectiveness in regard to MTC*s dual goal with related 
incidents and author comments. Topics in the second classification center 
on recommendations for improving review effectiveness in the future^ 

Before examining the relative "success" of erch type of reviewing, 

some concomitant philosophical issues should be explored. Two major 

quandaries plagued reviewers (and probably authors) throughout the course 

of critiquing a pi;oject*s courseware. What is^ the relationship between 

author and reviewer? What is the purpose of the review? Obviously these 

two questions are interrelated* Very often the rapport between author 

and reviewer determines the purpose or importance of the review* This is 

exactly the problem. In our experience^ neither the reviewer*s authority 

nor responsibility was ever clearly defined* As a result, reviewers 

generally felt that suggested lesson changes that were instituted were 

usually a re«ult of whatever powers of persuasion or cajolery they could 

\ 

summon* While the reviewer had no actual authority, s(he) usually felt 
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^^me ethical re^ii^erisibU ity for the quality of the l*^ssor Con<;eq,iont ^ 
feview£*rs found themselves continuutly ve^ei with the question or -vheLlier 
tht^ir re^poaj^lbility tended with aiding authors to meet the ciuthorV^ owri 
iitandartls or wiiether they (reviewers) shouLtl tr^' to entice tlie autiior into 
conforming to their standards- Usually they tried to aim sjomewhere 
between these two points, relenting in some areas while holding to others- 

Authors sometimes complained about the '^university approach" to 
lesson writing claiming that military students were "different" > so ^ 
certain iastructioaal approaches (increased iateractibn> open.-eaded 
questions^ etc.) were neither applicable nor feasible. It is true 
that there are differences between a military and university setting, 
Aberdeen authors were writing lessons for a widely varied group ranging 
from high school dropouts to college students, Also ^ th^ course materi^ll 
required very few cognitive skills^ centering largely on mechanical 
skills required to machine various sorts of keySj keyways> gear tect^V\ 
etc* As one author wrote > "It is a draiJtlc change to step from the I'niver- 
siLy of Illinois into an army classroom. ■ Idealism and the university 
apt^roach can be disastrous-'^ It is true .that some aspects of "techiiLCal 
ttciining^^ differ from acadeinicsj and reviewers gradually began to phase 
out coimnents in some area*;- However^ many- techniques MTC'suggested wliich 
were labeled ^'university approaches" were a<:tually strategies recommended 
or mandated by instructional design guidelines <^nd ruU^s for the individual 
^^rmed services. :^Oi^t so^-cailed "university approachc-'s" wcri.^ no^'er ^5hown to 
either succeed or faLl luntc author.<; often refused to try them say^ing* 
"That wuld never work with our Ktudont^;." After a wltile we began to 
ijftderetand that very oftc^n citing "the difference'' wvx*; r<'ally way of 

10. 
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raying, "1 tJon't likt; chat", or "I doii'c want co do t'nat." Tn one 
Ln:^triace reviewers suggoste'^l that authors at one AKPA svte try a 
vis^ual techniquo tliat h^ii originated at another ARPA site. In this 
instance al^o autliors at the first site claimed that their studeats 
were "difierent*' so Siyich a technique was neither "necessary" nor de&irab]o. 
Reviewers began to conclude that something other than '*tlie university 
approach'* might actually be tlie problem. Some authors even implied that 
the problem was a matter of de'fenslveness* of feeling that the second 
site was getting praise,>£rom reviewers whll^di the first was mairly getting 
criticism.* 

Consequently, reviewers felt trapped between two conflicting convic- 
tions. Wliile they believed ^that lesson quality was mainly the .author's 
responsibility, they also felt that failing to comment on a serious or 
recurring problem might be misinterpreted aa condoning or even encouraging 
something the re\;iewer in f act *rej;arded as ineffective or even detrimentals 

What Was Accomplished 

End^of-lesson reviews . Vlev^ed with the inevitable clarity of 
hindsight, the end-of-lesson review technique was almosf. a complete 
failur^^ both as an author aid and as a training tool. Authors mainly 
made only minor punctuation, spelling, or working chanSes> or codinj;^ changes 

(though reviews didn't usually stress coding revision). In a number of 

' 2 

instances even execution errors whicli were polrtted out 'in a review went 



Execution errors are programming errors so serious that the lesson 
will cease to function when the student gets to that point within the 
lesson. Typically* the jtudent is^then taken to a display ,tbat ^ays, 
"There'fi an erro/ in this leason. Try another one'* and then rerouted back 
to ^1 course index . ^ 
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unchanged. As one author wrote, "1 changed coding but fought most 
suggesLec" changes Co our insc rucc ional design plan.", l-Jhen asked what 
Che main adv^sntages of reviews were another author wote, "... help 
with coding — correcCions, a shorter, better way to code, ecc.^* One 
explanation is that coding changes are easier to make than strategy 
changes. As one author said, "We tend to get involved with the computet 
and forger, the desired resulc i training the students." 

Author comTiients about Che deficiencies o£.end-of-lesson reviews 
center on chree synersecic problem areas, First, authors commented that 
reviews were too detailed (a problem shared by in-progress reviews). 
One author commented "Your attention Co cfetail will destroy a new, 
marginal or struggling author." This partially supported our conclusion 
that the s^heer numbers of comments resalting from reviewing a leSson in 

its entirety could be devastating. A s^jscond and perhaps more serious 

^ . ■ 

pjfoblem was lack of communication that resulted not. only because of the 

divergence between "professional'* and "lay" people, but between civilian 
and military approaches^ One author commented, "Tour wording of ch^inges 
and new coding was very, very poor for those programming 1,000 miles away/' 
while another author wrote, "{ar, ez-military MTC member) was best able to 
review luurl lessons because he could speak our language. "The majority of 
the oth^r reviewers were good but the communication gap was real difficult 
to cress." Ye; contrary to this claitn,'even this .eK-military reviewer's 
comments went largely unheeded. A third, related theme that reverberated 
*througho^t nearly every ARPA project was the element of culture s'hock 
resulting from civilian*^^ and military personnel working in a cooperative 
effort. When asked to rate the importance of a number of areas included 
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tn roviL'Ws, one autlior v'rot<j with rt^terenct^ to vv>ntfnt jccunicv^ 'Tn our 
ca^ic you could not belp. ev^n in areas where you all were trained :,e. 
te^^t coni.tvuctiun. The idealisti*.' university ^ipproach prevailed." Siace 
one of tlit^ two (at thaz tin^e) ^f^C reviewers held a inaster's degree in 
ln<;truct ional deslw^ and sinr- many suggestions were taken directly 
from Air Force instructional desiga regulations* such a criticism did 
not seem valid. Purthermorej the r*=^vl€w recommendations werei once 
again, never tried and provr^n efEectlve or ineffective^ As one author 
said, "The authors aever gave the reviewers a fair chance/'- 

In'p^^o^ress reviews ^ In-prof;ress review techai^iues were considerably 
more successful in effecting short range lesson changes, Ijut only marginally 
successful as a long ranjje training device^ Authors instituted art average 
of 50-75% of the changes suggested in >frc reviews. These changes consisted 
raaialy of addiag graphics io replace or cJarify text, adding various Lypes 
of questions, improviag f^xistiog questions, distilliag wordy text, rear- 
ranging or breaking up ht:aviiy texced displays,' puaotuation, spelliag, 
etc* lite relative ''success" of ia*-progress reviewing Wus due ia part to the 
rapport built up through almost daily CQinmunication between site authors 
and ^tTC reviewers. The "success" of in-prog^ess reviews, however, also „ 
supports our ;=issuraption that ctiticism should be doled out a little at 3 
t i;iie . 

This is, of course, not to claim that ia^progress reviews; ate free 

r 

of snags. A number of issues still renain unsolved, for example, since 
there were only two ^f^C reviewers who performed in-progres.s reviews, it 
seemed to evolve that authors, usually nad lessons criti<^ued by the same 
reviewer each time Working on successive lessons together of course 
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resulted In authort; and reviewers building up n certain rapport- Ttms* 
the author came to know what sorts of comments that particular revxt'wer 
wa^ liable tci make, and the reviewer came to understand what sorts of 
comments would be accepted and wha^. would not* As a result some comments 
were eventually dropped entirely of at best categorized as perpetual 
debates, t'or example, one author and rt^vlewer continually debated whether 
the author' s\esson material actually belonged on PLATO or on a printed 
handout* 'The reV4^ewer' would say* "Just for the record, my old objection 
still stands", and the author would acknowledge the comment and then 
ignore it* Both knew dhat the suggestion would he made and that it would 
be ignored. 

Another norci, serious problem was the author dependency that sometimes 
resulted for rea?9ons which are not entirely clear even now. Some au'ihors 
relied heavUy on lestjoti reviews as the *;olt: "validation" of a les?9on, Tor 
example, In spite of repeated promptings, one author insisted on relying 
on reviewers to find problems In the lesson rather than running studencs 
through the lesson before It wafi given to actur 1 project students. V^hen 
actual students did take his lessens for the first timet they naturally 
found a number of errors that reviewers had not been able to anticipate*"^ 
Rather than proving the need for student te^stlng, hc^ ^jver, the Incident 
resulted in the author*s loss of faith in the value of reviewSi From his 
standpoint ,' rev u*wer credibility was at a fairly low level, for the remainder 
of the project. 

Not all authors responded in this manner, however. A number of authors 
made the comment, "Often when you say sdtnethlng in a lesson isn*t working, 
in fact the students object to It too. How do you know what they'll object 
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t^y* One -;uoh .aitijor did sec the nOc'd for sLudi'nt te^^tin^ to vaH<3atc 
U^-*^t>n^, ind in t.u-t to f;rt^it lon;^ths t** r^'<"ruir student^ Lo 

throtJ^it It ^^-t T^v KMor.* .it^tujl course studi'nt = ^tartc^d goln^ L^rou^;h Liu- 

Authors -liso reitt^rated lUe thent^ **Our students are difl"t"^rent than 
any otliers becau?>i? . * with .students eitlter l)eing vicictd as "too 
sniart** or "tuo dumb** for a suggested approacli. Sinee ln-proi;r£^*>s reviewing 
was lioned mainly on a paramt^dical project, the Variation reviewers heard 
mofit frequently was, "Our t^tudents even different from other military 
students because they're so highly motivated that all you have to do is 

r 

present the material and not realJy ivorry too much about how you do it. 
Sophisticated approaches aren't really necessary and only .slow them down/* 
Ulipn three of the original sixteen students (an alarming rate for thisi 
paramedical courise) flunked out of the paramedical program entirely and 
the remainder of the s^tudents asked to b^ reassigned to the regular class- 
roon, the theme bt;came, **These students are so poor that 'sophisticated 
approaches' are beyond tiiem.'* for example, a comrflon reviewer comment was 
that" questions were aimed at too low a level of learning, requiring the 
student to demonstrate simple recall rather than apply a concept to a 
new situation. Here again, review comments conformed to actual Sheppnrd 
course document requirements which specified the need for teaching to apd 
testing! at the highest cognitive levels ( anrslys i s ^ syiithes i s f e va 1 ua 1 1 on y 
etc.) ovt^r recall or restatewnt. At tiie beginning or the project, some 
authors claimed that "sophisticated approaches*' took u|> the student's 
valuable time. Later, after r^*viewers again made the same objection to 
/i~jmnibe,r of later lessons, tHe same authors retorted that "these" students 
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were at too low a levxl t'or "sophij^t icated" questions, and l>ad to be 
"spoonfed" the material. Some authors also admitred that part of the 
problen was ti»e difiiculty in coding shurt rtnsv:er types oi quej^tions, 
tiiough a full-time programmer was on staf t _pt tlie site. 

Over rhe cour^sc of che project, OTC reviewers observed a number 
of author b f^havi-Trdl chan^t s, vithout apfarenc corresponding attitudinal 
changes, lor exampie, one author consistently resisted writing student 
objectives for his lessons. In a number of consecutive reviews, the 
reviewer reiterated the usefuinesi> of specific student objectives in 
a^^isisting the author in knowing What material to iticlude, how tlie infor- 
mation s;hculc! be organized, what he actuaily expected from the student and 
how he was going to test whe^ther tht^ ^^tudeni iiad learned what he intended, 
etc* After a uhiU * the author finaily bej»an to vrite objectives, at 
tinted t^'Ven submitting objectives for revit?w apparently before the lessen 
had even been si::jrted, Vhe reviewer a>;sumcd thi*; represented a change 
(or rather an addition to) the author's 3n»5tructional philosophy. How" 
ever, wlien it wa^ pointed out that die objectives hiid content of one of 
his lessons didn't niatch he replied, *'0k, I'll write tlie objectives over 
again*" The thouj^ht that it was simpler to jui>t rewrite '-he objectives 
>;howed a lack of understanding of what the purpose ot i'tating object ive>; 
really wajj. 

In another instance, an author and reviewer debfited at some lenj^th 
the value oi interspersing; the lesson with interactive questions in 
reinforcing the lesson material, giving the. student practice at utilizing 
information he would later be tested on, "monitoring" the student's 
progress, etr.. One of the sources of dissagreeroent was in their conflicting 
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joi-«ltion'^ oi "interai.-tlcn''- /he reviewer defined interaction ^ny 
vittKUi^Mi in u!uLh the ->tudont is required to respond by writing a 
^omnJtnt^ .^tiKini^ i qut*;tion or giving an answer- The author defined inter- 
action .iS any ^iqdent input whr^tsoever including simple keypresses (NEXT, 
BACK, etc.). Til*; author finally bo^n to put "interaction" into htii ' 
leifsons, liut since his definition was so broad, dimply Increasing th^ 
number of student inputs did not appreciably r^tse the quality of his 
lessons^ t\lso, what questions he did include \fere mainly y^^/no or 
copyframe types in which the student simply scanned tlie page for the 
correct answer and then "recorded" it. In some cases, questions wete 
not even at the le^^el of simple recall. It seemed app^irent that he was 
including interaction strictly for, interaction's salte . 

In anocher episode, a number of authors ^t one site were producing 
.';trlctly linear lesjions with all, students routed tivrough the same path- 
UlUle a Huear format is certainly not inher^^ntly detrimental, tH'e 
reviewer tried to persuade authors to consider including at least a 
small number of lessons in which the student had more autonomy. Indexes, 
''crossroad" type choice pages, etc. vrere suggested. Joon the revlevrer 
began to see a flurry of index pages appear. However, students were 
still instructed to go through the lesson in the order given* In out case, 
while a lesson index was supplied, it seemed in fact to be more a table 
of contenti> ^ince the student was never allow^d^ to choose wher'i to go. 

In all these instances, it is apparent that, while external changes 
were certainly made, the attending internal changes never crystallized. 

In none of the preceding incidents is it clear whether the problem 
\fas the reViewerc' collective lack of persuasive ability, lack of 
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coininunication^ or simply that, rather than J^bate the Issues any longer, 
tifUthors chose to make token ch:inges. In n number o£ instanc es real 
changes were mjde. For example, a number o( authors at one site be;;an 
to eliminate "copyframe^' questions alno>:L entirely from their lessotis^ 
learning instead to i>Tite c^utistions .linked at liigher levels of learning. 
In many cases, a^jwever, it seemed ^ipportiu that, though juthors' approache=5 
sometimes chanj^ed, the attending attvitudio^i; changes never mctainorphosed. 

What Can Be Accomplished 

In spite of our sometimes frustrating experiences, we feel very 
strongly that some sort lesion review is essential not onlv for 
int?xperienc*id authors but aliio for already established authors. In the 
Satne way that Lhe publishing industry exercises editorial standards, 
CBE ne^ds some quality control metho^is to ivisure at least luinimum lesson 
stancJards. Reviewing can be a valuable tool for lesson development, 
3 "soft" Step between proi^ramming and fecdbiick from student runs. Based 
on our ^;iperiencei; and perceptions, and \;ommonts from^authors rit various 
sites, we feel confident in making the following recommendations. Wliile 
nuarly all our testing and theorizing has been (Jone with the Tnilitary^ 
many suggestions should apTf>ly with t;qual validity in any number of situa- 
tions. 

After comptlin^^ and *;orting throu^^h past recol Itsctions and comments, 

three major areas of concern standout 

— reviewer characteristics 
— reviewer/ author relat ionship 
. ^ — review content/format 

The following discussion deals with suggestions in all three areas. 

17 
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R eviewer Charact<sristics 

In a resounding expression oC unanimity, all authors agreed that a 

lesf>on reviewer should be on site/ This is perhaps the strongest and most 

widely expressed recoounendation we will make. As one author wrote, 

Of all the reviews that I had, I felt the Giosc useful on^ was 

the one which I had side by side with when' she was down < 

here* 'I felt 1 learned more in that 2-3 hour period than in 
all the oth^r reviews* There were alot of cUings she said and 
did chat just wouldn't have been included in a review. Reviewers 
should have come down to deliver the reviews personally or to 
do the reviews right there. I'm not sold on the idea that you 
can de reviews long distance. 

Some authors also felt that a reviewer should have teaching experience, 
though ^there was wid^ disagreement over whether the reviewer should also be 
a subject-matter expert. Most authors^eemed to chink that subjecc- 
macter expertise was not necessary. One author even said, "Somecim^ss 
it was good chac che reviewer didn't have subject-ma^cter experience. Ic*s 
coo easy (for a subjecc-maccer expert] to miss some things . . * an outsider 
can ask 'what's going on here?'" Si^ce roosc sites are gt-xerally staffed 
with a number of subject-matter authors, we feel no real need for the 
reviewer to be a subject-matter specialist, coo. A number of non-ARPA 
sices {as well as one nocable aRPA sic^) have observed the tendency for 
subject*-tnacter experts to "review" lessons solely on the basis of concent 
accuracy. In fact, some projeccs and autlwrs specifically requested 
subjecc maccer reviews only, scacing explicicly chat comments on t?resenca- 
tion, instructional strategy, etc. are of no interest. This is certainly 
a valid area, but by no means the breadth of what a lesson is or the only 
cricarion for lesson effectiveness. Though some content specialists 
would disagree, concent expertise does not presuppose instructional exper-^ 
cise* By che same token, lack of content experience need not imply lack 
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of credibility, A l^s^son needs to be crit LqueJ ^from <i number of dil l ^'rent 
perspectives, one of which is from tlie iiititructional design point ol view. 
Thus, Just ais it's reaj;oriable/ to ask a ctmteiit expert to review th : fontent^ 
it's also reasonajsle to rely on an instruct ional designer to review Llie 
instructional approach, 

A more illusive but equally Importanc quality is reviewer personality. 
Nearly all authors agneed that a reviewer raust be able to "handle" a 
variety of author temperaments. Typical author comm^^ts are, " , , , it's 
very tough to find the right individual .who can handle each author indivi- 
dually and appropriately", and "Personality requir(^nt*nts vary with a^jthor 
tejmperanent A reviewer needs to be somL*thing of a psycho lofiii>t , ai)le 
to employ various approachos, For a com^^lete description oT rt^viev^ 
techniques utJcd tltroui;hout t)ie >rrC project see Lesson keview (irniciiSt 
Goldstein ^ CaU-Himwich, 1975). 

Reviewer/Author Relationship 

Many autiior^ (and reviewL*rsJ felL that the success or failure of a 
review was a direct reijult of the relationsliip between author and reviewer, 
A« one author said, "it's all so personal. It's alt in the relationship 
betu^een the reviewer .ind author,'* Another author stated, "Rapport is 
important , it's intangible, but it efft*ct5 the results." Rapport 
will always be a factor in any cooperative effort. However, the hejvy 
reliance on rapport in reviewing is often (Counterproductive. A number of 
step$ can be taken to reduce the iiDportance of the subjective relationship 
between author and reviewer c.nd to clarify the purpo,se of the review 
Itself, Firsts If the reviewer is to be in any way responsible for the 
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quality of the Itsson*; produced, {s)he shouid be accorded some authority ^ 
at the outlet by the funding aj^ency or project management. A "confiultant" 
position Ls nut a fitronK vantage point' from which to effect change. In 
any project (particularly in the military), if an on site reviewer is a 
member of the military, hie/her rank should^ be equal to that of the highest 
ranking author. At one ARPA site, the only instructional design specialist 
was an enlisted man whilo all but one of the remaining authors were officers 
His suggestions were not well received, and were in fact either totally 
ignored or never requested at all. If tlie reviewer (either on br off site) 
is a civilian working with military authors, his/her perceived authority 
must be equal to that of the authors, ihis, of course, contains the seeds 
of a reverse sort of problem. If the reviewer has higher authority, some 
authors may follow the letter rather than the spirit of suggestions to 
give the impression of obeying a superior. In the lorfg run, the most 
impor*:ant factor may still be the relationship between author and reviewer. 

If the reviewer is off site, his/her credibility can be greatly 
enhanced by frequent site visits at wiiich times (s)he can become familiar 
with authors problems in the environment in which they must function. This 
would also help in dispelling the "ivory tower** syndrome. For example, 
two authors ac one ARPA site repeatedly discounted portions of what ^^^C 
reviewers suggested, saying, ".You don't understand. You don't understand 
our students or our problems." When a reviewer finally visited the site, 
both authors commented, "Uow that you've seen our students, you won't have 
any trouble reviewing", though the reviewer had in fact only literally "seen' 
the students in hallways, etc. rather than in a scheduled PLATO clJiss. 
Though the reviewer felt there was little discemable difference in the ^ 
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comments she made in successive reviews* both authors seemed to be 
satisfied that now her reviet*s were '^better*** How much credibility 
exists in fact and how ntuc|i in the eye of the beholder is sometimes 
speculative. If the problem exists, however (and it usually does), 
debating the p^oint will never solve it* 

A second step toward clarifying the purpose of the review would be 
to contract at the outset for specific feedback the^autbor would like 
from a review. This is particularly important if the reviewer is seen 
^ only as a consultant. To facilitate this and to establish some commoTi 
instructional ground, the author and reviewer should also meet before 
even the first review is requested and try to establish instructional 
design concensus. This would also afford each an opportunity to get a 
better feeling for the other's biases, nomenclature, etc. 

Review Format/Content 

One of the most Important recommendations we could jnake is the 
establishment of on-^slte* peer reviews as a regular part of the workings 
of a project* This could take the form of regular group meetings* or 
individual, more informal one-on-one types of reviews. At every site at 
which peer reviews were a regular routine, authors said they relied 
approximately half on MIC reviews and half on peer reviews both for 
feedback and as a tool for revision. In order to effectively review 
each other's lessons, however (i.e. comment on more areas than spelling, 
punctuation, content accuracy* etc.), authors also need more explicit 
training in instructional design. 

If possible, reviews should always be performed face-to-face with the 
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rtivLewer invol-ved in the lesson from the planning stages on. If 
revietif^ can'i be clone in person, on-line monitorinj* or even over-the-phone 
revLewing are good .litornative.^ jnd preferable to end-of--lessoii or 
hard-copy-only type revU-ws. 

A lesson iihould also be revieued in segments. We found this to be 
a highly effective deterent to the buildup of author def ensiveness or 
subj ectivity. 

Finally, depending on the contracted specific feedback, reviews 
should stress organization, alternate instructional approaches, clarity 
of text, quality/quantity of interaction, etc. Since enumerating mis- 
spellings and grammatical errors both is time consuming ancl increases 
the number of perceived "criticisms" in a lesson review, the lesSon should 
already have been proofread for such incidentals before a review is 
requested. 
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^'\ppenciix A: Auchor.Atcicude Questionnaire 



^£\\e toUowing'is a list of suggestions you may or may not have been 
looking for in an MTC review. Please indicate the importance of chose 
items you VANTED by ui>ing the racing scale provided. Items you were 
NOT interested should rate as 0. 



4 — wanted and very important 

3 — wanted and important 

.2 — wanted and moderately important 

1 — wanted and minimally important 
0 ~ EJOT wanted 



_a. grammatical usage, spelling pun^ccuacion, etc. 
_b. quality/quantity of student interaction 



_^c. appropriateness of cone and style 
d. clarity of -explanations 



^ effectiveness/arrangement of graphic displays 

f. content organization 

g. content accuracy 

^h. alternate instructional strategies 

i. coding efficiency 



J. other (if other, please specify) 
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2. Using the following scale> please rate the importance in lesion 
development of the sources of .informat Ion listed below. 

7 

4 — very important 
3" — moderately important 
2 — minimally important 
1 — not important 

0 — don' C know ' 

a. local review by project member(s) 

b. review by OTC 

c. review by instructor/users. 

^d. review by outside experts 

e. observation of students 

C. comments or questionnaire responses from students 

" student data (on-line) 
h. test or quiz result?; ^ 
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3* What wete the main Jisadvancages of MTC reviews? 



4. were the main advantages of MTjC reviews? 



V 



5. t regarded the MTC reviews I received as: 

a. Generally a waste of time 

b. Interesting but not very useful ' 

c. Moderately useful 

d. Very useful , 

6, In what way, if any, did you change your approach to lesson writing 
based on the information you received in MTC reviews? 



How would you characterize the ''ideal'V reviewer with respect to; 
subject-matter knowledge, personality, geographical location, 
teaching experience, instructional design experience, etc.? 



\ 
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